DOCUMENT RESUME 



ED 380 984 FL 021 930 

AUTHOR Steel, David; Alderson, J. Charles 

TITLE Metalinguistic Knowledge, Language Aptitude and 

Language Proficiency. 
PUB DATE Mar 94 

NOTE " 14p.; Paper presented at the Annual Language Testing 

Research Colloquium (Washington, DC, March 5, 
1994) . 

Reports - Research/Technical (143) — 
Speeches/Conference Papers (150) 

MF01/PC01 Plus Postage. 

College Students; Comparative Analysis; Foreign 
Countries; 'Trench; Grammar; Higher Education; 
Introductory Courses; ^Language Aptitude; ''Language 
Proficiency; Language Tests; '''Metal inguis t ics ; 
*Psycholinguistics; Second Language Learning; 
Statistical Analysis; Testing 
University of Lancaster (England) 



This paper presents a preliminary investigation of 
the psycholinguistic abilities of first-year undergraduate student 
learners of French. In particular, it reports on the construction and 
use of a battery of tests of metalinguistic knowledge, language 
aptitude, grammatical accuracy in French, and French linguistic 
proficiency, and explores the relationships among these measures with 
a view to establishing levels of metalinguistic knowledge in incoming 
undergraduates. The paper reports on pilot work and makes proposals 
for further research in the light of the results. Contains three 
references. (Author) 



PUB TYPE 



EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 
ABSTRACT 



ftftft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft 

* Reproductions supplied by EDRS are the best that can be made ' v 

* fro" 1 the original document. " 

ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ■< X ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft 'V ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft 



-3- 

oo 
as 

o 

CO 

cn 

Q 
w 



Metalinguistic Knowledge, Language 
Aptitude and Language Proficiency 



David Steel and J. Charles Alderson 



■ PERMISSION TO REPRODUCE THIS 
MATERIAL HAS BEEN GRANTED BY 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC)." 



U $. OffARTKlNT Of EDUCATION 
OtTtce of Educational Raaaarch ami Improvafnant 
EDUCATIONAL RESOURCES INFORMATION 
CENTER (ERICI 

Vi'hia document hat Oa «" faptoduced aa 
'•caivad Irom lha pataon Of organiiation 
"originating il 
O Minor chungat nava oaan mada to improva 
reproduction Quality ^^^^^^ 

a Pointa ol view o- opiniona etatad m thia doeu- 
man' do not necaaaariiy reoreeeni oHicial 
OERI poaition or policy 



9 
7i 

0 



o 

ERIC 



1 

METALINGUISTIC KNOWLEDGE, LANGUAGE APTITUDE AND LANGUAGE 
PROFICIENCY 

David Steel and J Charles Alderson 



This paper presents a preliminary investigation of the 
psychol inguist ic abilities of first-year undergraduate student 
learners of French. In particular, it reports on the 
construction and use of a battery of tests of metalinguistic 
knowledge, language aptitude, grammatical accuracy in French, 
and French linguistic proficiency, and explores the 
relationships amongst these measures with a view to 
establishing levels of metalinguistic knowledge in incoming 
undergraduates. The paper reports on pilot work and makes 
proposals for further research in the light of the results. 



1. The Context: 

First-year undergraduate students of French at Lancaster 
University 

2. The perceived problem: 

According to university modern language teachers, incoming 
students know little about language and are less accurate in 
their language use than previous years' students. However, 
first-year language teaching assumes both knowledge about 
language and a degree of grammatical accuracy. 

3. Issues: 

What is meant by 'knowledge about language' needs to be 
explored, but it typically includes a knowledge of and ability 
to use metalanguage appropriately. 

Staff impressions of these deficiencies need corroboration, 
and no empirical evidence was available. 

The consequences of such deficiencies may be that students 
need to be taught a metalanguage in secondary school. However, 
it may be that university teaching should change instead, in 
order not to rely upon (non-existent) metalinguistic 
knowledge A further possibility, however, is that 'knowledge 
about language' and language proficiency are unrelated. 

4 . Background : 

1. Communicative language teaching in schools has led to 
greater ability to use the modern foreign language, but a de- 
emphasis on accuracy and metalanguage. 

2. Knowledge about language is held to be an important 
component of education in general, and is thought to 
contribute to language learning. 
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3. Language aptitude, one key component of which is 
grammatical sensitivity, is still widely regarded as important 
in language learning success. The relationship between 
aptitude and knowledge about language is unknown. 

4. Bloor's 1986 survey showed low levels of knowledge about 
language, even for 'linguists', but had no information on the 
relationship of such lack of knowledge to language 
proficiency . 

5. Research issues 

In light of the above, several possibilities present 
themselves : 

1. University foreign language students' supposed lack of 
metalinguistic knowledge does not exist, and the situation now 
is in fact no different from previously. 

Alternatively, students' metalinguistic knowledge has indeed 
declined and is now at a low level. 

2. Even if this suspected lack of knowledge can be established 
it does not bear any relationship to competence in the modern 
foreign language. 

Alternatively, metalinguistic knowledge does bear such a 
relationship. 

3. If students can be shown to lack the knowledge presupposed, 
the consequences are that they need to learn it at university, 
and current assumptions concerning the appropriate methodology 
for first-year language teaching will need to change. 

Alternatively, they do not need to learn it at university but 
the methodology of teaching language at university will need 
to change to avoid the use of metalanguage. 

4. In language proficiency, a formal metalinguistic knowledge 
of grammar is less important than the ability to detect 
systematic and meaningful patterning in language: language 
aptitude. 



6. A pilot study 
6 . 1 Aims 

This paper reports on a pilot study designed to begin to 
explore these and related issues. The aim of the research was 
to establish levels of 

1. the metalinguistic knowledge, applied to English and 
French, of university students of French on entry. 

2. their accuracy in French grammar. 
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3. their aptitude for learning language 

4. their proficiency in French, as involved in understanding 
French texts 

5. their sel f -assessment of their linguistic abilities in 
French 

and the relationships among these variables. 



6 . 2 Instruments and data collection 

A battery of tests was developed, comprising: 

a) a 100-item Test of Metalinguistic Knowledge, for both 
French and English. (Part of this test was based, with 
permission, on the instrument used by Bloor, 1986a, in order 
to facilitate comparison with his results,; other sections 
included metalanguist ic items identified by staff of the 
French Department.) 

b) a 45-item Test of Language Aptitude (Part IV of the MLAT, 
Words in Sentences) 

c) a 100-item Test of Grammatical Accuracy in French 
(consisting of items considered important by staff of the 
French Department, see below). 

d) a questionnaire asking students to self-assess their 
ability to use French in a range of settings 

e) a 50-item standardised Test of French Reading Comprehension 
(constructed by CITO, the Dutch National Testing Agency). 

In addition, a bio-data sheet was constructed to collect 
information on potentially important background variables such 
as : age; sex; length of time spent learning the foreign 
language; experience of learning other modern or classical 
languages; time spent in the country of the target foreign 
language; parental example (whether native speaker or teacher 
of target language); performance on measures of language 
competence at A-Level . 

This battery was pre-te.sted on Sixth Year secondary school 
pupils studying A Level French. The trialling provided 
valuable information on timing and level of difficulty, and in 
the light of the trials, the tests were revised. All first- 
year undergraduates entering the French section of the Modern 
Languages Department at Lancaster University took the test 
battery in a two-week period in October 1992 (not all 
students took every section of the battery: see reported n 
sizes). In addition, students studying first-year introductory 
courses in linguistics took the Bloor part of the 
Metalinguistic Knowledge test, to enable comparisons with the 
Bloor data and with the results of the French undergraduates. 
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6 . 3 Results 



The descriptive statistics are as follows: 

Table 1: Descriptive Statistics 





Metal ing 


MLAT 


Grammar 


French 










Reading 


Number of students 


128 


128 


107 


102 


Number of items 


100 


45 


100 


50 


Mean score 


59.88 


22. 14 


54.83 


31.78 


Mean as % 


60% 


49% 


55% 


64% 


Standard deviation 


11.71 


4.99 


11.10 


8.56 


Reliability (alpha) 


.897 


.687 


.856 


.890 



Reliabilities are entirely satisfactory for this test battery, 
except for MLAT. This latter finding is somewhat surprising, 
as the test is standardised and has been widely used for 
thirty years. 

The descriptive statistics reveal that the tests were all of 
appropriate difficulty for this population and resulted in a 
reasonable spread of candidates across the range of possible 
scores. Item analysis revealed a range of item difficulties, 
as expected, but virtually no items were unsatisfactory 
because of negative discrimination. It was concluded from the 
item level data that the tests were satisfactory. 

Students were 'weakest* on the Aptitude test, and 'strongest' 
at reading in French. However, the spread of students on the 
reading test was quite large: students vary most in their 
reading abilities. The mean score on the Grammar test of 55% 
suggests that, if this test reflects what students are 
supposed to know when they arrive at University, their 
knowledge is indeed weak. Similarly, if students could have 
been expected to 'know' the metalanguage in the Metalinguistic 
Knowledge test, their performance is worrisomely poor. 

6.4. Comparison of performance on the Test of Metalinguistic 
Knowledge of students of French, introductory linguistics and 
Bloor's original data 



Table 2: Comparison of Lancaster 1992/Bloor 1986 data 



Lancaster French (n=l28) 



62.8% 



Blcor 'Linguists' (n=63) 



77.5% 



Lancaster 'non-linguists 



50% 



Bloor 'non-linguists' (n=175) 



51.1% 



6.4.1 Lancaster students: French students vs students of 
introductory linguistics 

Levels of metalinguistic knowledge varied considerably in the 
sample, although all groups do well on a few items 
(identifying vowels, verbs, nouns, adjectives and subjects). 
All groups do particularly poorly on some items and rather 
badly on most items. 

The French group is notably better than the other two on 17 
out of the 24 questions. The differences Bloor found between 
linguists and non-linguists were broadly reflected in the 
differences between French students and students of 
linguistics. It seems that learners of French have a better 
metalanguage than other students. (Some might even wish to 
argue that the study of a foreign language increases one's 
sensitivity to grammatical categories, and hence performance 
on this Test of Metalinguistic Knowledge. However, the 
research was not designed to enable us to investigate this 
suggestion. ) 

Nevertheless, it would appear that any instruction that 
assumed that first year undergraduates knew much more than 
'verb', 'noun' and, possibly, 'adjective' would cause students 
difficulties . 



6.4.2 Comparison with Bloor data 

The differences Bloor found between linguists and non- 
linguists were replicated at Lancaster, but when Bloor' s 
linguists were compared with the Lancaster French students, 
the latter were generally weaker. However, when Lancaster non- 
language students are compared with Bloor' s non- linguists, the 
differences virtually disappear. 

Clearly more research is needed, with larger populations from 
other universities, but it would appear from these comparisons 
that, assuming comparable ability levels on admission for the 
two institutions, the metalinguistic knowledge of students of 
French has declined. However, non-linguists remained at 
roughly comparable levels of metalinguistic knowledge (which 
is considerably lower than language students in most 



6 



linguistic categories). If the populations are equivalent then 
this might suggest that overall standards have not declined, 
or that they cannot go much lower. Rather, what has declined 
is the metalinguistic knowledge of students of French, 
relative to the rest of the student population. 

We again conclude that any instruction that assumed that first 
year undergraduates understood much more than 'verb', 'noun', 
'subject' and, possibly, 'adjective' would cause students 
difficulties. 



6.5. Students' Knowledge of Metalanguage and Grammar 
6.5.1 Metalanguage. 

The Bloor test was only one part of the Metalanguage Test. The 
latter includes a self-assessment of one's familiarity with 
grammatical terms, tests of knowledge of English and French 
grammatical terminology, and the ability to use the terms to 
identify parts of speech in French texts, as well as Bloor' s 
test of ability to identify parts of speech in English text. 
An interesting comparison is sub-test 6 - the Bloor items in 
English - with sub-test 4, which covers similar items but 
applied to French. The two lists of form classes to be 
identified in French/ English texts partially overlapped, and 
the results for overlapping items were: 

Table 3: Bloor items: French/English 





French 


Engl ish 


auxiliary verb 


61% 


54% 


adjective 


90% 


91% 


infinitive 


97% 


73% 


preposition 


69% 


62% 


indefinite article 


25% 


34% 


past participle 


91% 


91% 


conjunction 


41% 


56% 


finite verb 


49% 


41% 


Mean 


65.4% 


62.8% 



These results show a tendency for the English itsms to be 
easier than the French ones. Although the differences are not 
marked, the results suggest that being able to apply the 
grammatical term to English text does not guarantee an ability 
to apply the same term to a French text. 

Sub-test 1 asked students simply to indicate whether they were 
or were not familiar with certain metalinguistic terms. Of the 
30 items, only 10 were reportedly familiar to more than 80% of 
the students. Particularly unfamiliar were: 'predicate', 
'antecedent', 'partitive article' 'relative clause', 
'subordinate clause', 'relative pronoun' and 'transitive 
verb' . 
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Comparisons are possible between items students claimed to be 
familiar with and their performance on those items elsewhere 
in the test. The results are: 



Table 4: Claimed familiarity/ actual 



conjunction 
direct object 
indefinite article 
indirect object 
infinitive 
noun 

passive voice 
past participle 
predicate 
preposition 
proper noun 
relative clause 
transitive verb 



Claimed 


Actual 


familiar 


correct 


72% 


41/56% 


95% 


46/67% 


59% 


25/34% 


93% 


61% 


98% 


97/73% 


98% 


99% 


74% 


48% 


100% 


91/91% 


7% 


8% 


91% 


69/62% 


69% 


78% 


35% 


17% 


42% 


12% 



In general it appears that students considerably overestimate 
their familiarity with these grammatical terms. True 
unf amiiiarity is likely to be greater than the results for 
sub-test 1 suggest. 

If teaching is based upon an assumption that students know the 
terms tested in the Metalinguistic Assessment Test, those 
assumptions, and possibly that teaching, need to be revised. 



6.5.2 Grammar Content sub-tests 



The French Grammar test covers a range of different structures 
and includes a variety of different methods of testing, 
including a gap-filling test, a translation test, and several 
transformation tests. Items tested include: gender, plurals, 
adverbs, special adjectives, superlatives, pronouns, passe 
compose, relative clauses, verb tenses and possessive 
adjectives 

The most difficult section proved to be sub-test 9: relative 
clauses, with very weak performance by students. The easiest 
section by far was sub-test 7: Passe Compose (form). 
Candidates varied considerably in their abilities on all 
sections of the test, which discriminated well between weak 
and strong. If it is thought that students ought to do well on 
this grammar test, the fact that all means except one are 
below 70% indicates considerable weakness, but the high 
variability in performance means that some students do very 
well, whilst others are very weak, on all sections except the 
gap-f i 11 ing . 

In short, students vary greatly in the accuracy with which 
they use French. The interesting question is whether the 
ability to produce accurate French correlates with a knowledge 
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of metalanguage, or with grammatical sensitivity as measured 
by the MLAT, or indeed with an ability to understand texts 
written in French. 

6.6 Students' French Comprehension Ability 

The results in Table 1 showed the comprehension test to have 
been the easiest of the battery, but not too easy, with a mean 
of 64% and a good spread of scores. What is clear from an 
inspection of item facilities, however, is that this 
discrimination was a result of speededness: 78% of students 
failed to get as far as the last two items. 

Students did well on the items for the first four texts: most 
facilities are over 75% (and 5/33 over 90%). In other words, 
when students did read the texts and respond to the questions, 
they tended to do so correctly. The problem seems to be that 
many students read rather slowly. This is interesting in that 
it suggests that students do understand written French even 
though some read faster than others. Are the slower readers 
those who have most problems with grammar or metalanguage, or 
who have less 'aptitude'? 

6 . 7 Relationship among tests and constructs 

In order to explore the relationship among the tests, 
correlations were calculated. For the 92 students who took all 
the components of the battery, the results are as follows: 

Table 5: Intercorrelations among tests 

MLAT Grammar French Reading 

Metaling .435 .429 .072 (NS) 

MLAT .203 (NS) . 186 (NS) 

Grammar .399 



These results seem to show that the test of French Reading 
shows no relationship to knowledge of metalanguage, or to 
language aptitude as defined by MLAT. The Metalanguage and 
Aptitude tests, however, do show a moderate inter- 
relationship, albeit less than might have been expected. The 
Grammar test relates moderately to the Metalanguage test, but 
also to the test of French Reading. 

To explore the relationships further, a factor analysis was 
conducted, with Varimax rotation. 

Table 6 

Varimax rotated factor analysis: Two factors, Eigenvalue > 1.0 

Factor 1 Factor 2 Commonality 
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Grammar 

French Reading 
Metal ang 
Aptitude (MLAT) 
Eigenvalue 
Variance % 



394 
039 
869 
782 

.88 
47% 



.721 
.906 
. 120 
.096 
1.01 
25% 



675 

,823 
,770 
,620 



This analysis shows that there are two factors underlying 
these four tests: a metalinguistic factor on which 
Metalanguage and Aptitude tests load high, and a test of 
French Reading. The Grammar test is split across the two 
factors, but with a higher loading on Factor 2 (French Use?) 
than on the metalinguistic factor. 

Who the analyses appear to reveal is that the Metalanguage 
test measures something different from what is measured by the 
Grammar and Comprehension tests. As was shown by the simple 
correlation matrix, metalinguistic knowledge and aptitude are 
related, although not closely. However, there is no 
relationship between linguistic proficiency defined by the 
tests we used, and metalinguistic knowledge. Subsequent 
sections will explore this further by comparing metalinguistic 
knowledge with self -assessed proficiency and with proficiency 
as measured by A-Level grades. 

6 .8 The relationship between self -assessment and test 
performance 

There is little interpretable relationship between the 
difficulties that students predict for their use of French in 
a number of different communicative situations, and their 
ability as measured by the four tests. Students who predict 
few difficulties do not do better on the tests, nor vice 
versa . 

However, students' assessments of their global ability to use 
French in the four skills did associate significantly with 
their test scores. 



Table 7: 
(p<.05) 



Association of self-assessed skill with test score 



self assessed skill/ 
test 

metalanguage 



reading 
writing 
listening 
speaking 



Yes 
Yes 
Yes 
No 



aptitude grammar comprehension 



No 
No 
No 
No 



Yes 
No 
No 
Yes 



Yes 
No 
Yes 
No 



Table 7 summarises these results, showing that the Aptitude 
test shows no relationship with self assessment. 
Interestingly, the Metalanguage test shows the most consistent 
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associations with perceived ability in the skills: with the 
exception of speaking, those who rate themselves high on a 
skill tend to score high on the Metalanguage test, and 
conversely those who rate themselves low tend to score low on 
the test. 



6 . 9 The relationship among biodata variables and test scores 

Crosstabulations were run between biodata and test scores, and 
some interesting relationships were found. See Table 8 below: 

Table 8 

Associations between bio-data and test scores 





Metaling 


Aptitude 


Grammar 


Reading 


Language Major 


No 


No 


No 


Yes 


Grade A level French 


No 


No 


Yes 


Yes 


Sex 


Almost 


No 


No 


No 


# weeks in France 


No 


No 


Yes 


No 


Type of sec school 


No 


No 


No 


No 


Other A Level Language 


No 


No 


No 


No 


Grade in above 


No 


No 


No 


No 


A Level EngLit 


No 


No 


No 


No 


A Level EngLang 


No 


No 


No 


No 



When ' Ir tended Major' was recategorised as ' Language- related 
subjects 1 and 'non-language subjects', there was no 
relationship between this variable and scores on the 
Metalanguage, Aptitude or Grammar tests, but there was with 
the Reading test: students intending to study languages 
understand French texts somewhat better, but do not do better 
on Grammar tests, nor have they better aptitude, or higher 
metalanguage scores 1 

Significant associations were found between A Level French 
grade and scores on Grammar and Reading tests on the one hand, 
but not with the Metalanguage or Aptitude tests on the other: 
students who do better at A Level do not necessarily have 
better grammatical sensitivity or knowledge of metalanguage, 
but their accuracy in French grammar and their French reading 
comprehension are better. 

Interestingly, sex was almost significantly associated with 
Metalanguage scores (p=.051), but not with Grammar, Aptitude 
or Reading. Number of weeks spent in France was not associated 
with Metalanguage, Aptitude or even Reading test scores, but 
it was associated with Grammar test scores. No significant 
associations were found between type of school attended, and 
any of the test variables: the school students attend does not 
make a difference to the test scores. 
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Whether or not students took another language at A Level has 
no significant association with any of the test scores, nor 
does the grade achieved in that other foreign language: doing 
another language does not increase one's metalinguistic 
'ability', aptitude, accuracy in French grammar or 
comprehension of French. Nor is it the case that doing well at 
A Level English Literature or English Language will be 
associated with better scores on any of the tests. Whatever A 
Level English (Language or Literature) measures, it does not 
relate to grammatical sensitivity or knowledge of 
metalinguistic terminology. 



6.10 Summary and preliminary conclusions 

The correlat.ons between the different components showed only 
moderate relationships: between metalinguistic knowledge and 
French grammatical accuracy (.43) and metalinguistic knowledge 
and language aptitude (grammatical sensitivity) (.44). 
Proficiency in reading French showed no correlation with 
either aptitude or metalinguistic knowledge, and only a 
moderate correlation with French grammatical accuracy (.40). A 
factor analysis revealed two separate factors: metalinguistic/ 
aptitude and French proficiency. 

Metalinguistic knowledge showed significant associations with 
self-assessed global proficiency in three of the four skills, 
but not with other self-assessed components of ability. 

Metalinguistic knowledge failed to correlate with scores on 
the French A-Level exam (arguably the most comprehensive 
measure of language proficiency we have). 

Tentative conclusions are that students can be proficient in 
the use of French without having aptitude defined as 
grammatical sensitivity, and without high levels of 
metalinguistic knowledge. Accuracy in French contributes only 
moderately to this ability to use French, at least for the 
purpose of understanding written texts. Although 
metalinguistic knowledge showed some relationship to accuracy 
in French, its contribution to proficiency defined by a 
reading test, by A-Level results and by self -assessment seems 
to be minimal . 



7. Limitations and further research 

The results of the pilot study revealed a need to refine the 
test instruments. For tests of French accuracy, one 
possibility might be to include methods like editing tests, 
multiple choice tests, even tests of writing or translation. 

The range of cests of linguistic proficiency should be widened 
from tests of reading to include tests of listening and 
writing ability. 
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The Test of Metalinguistic Knowledge needs revision to remove 
inefficient items, and to improve content coverage. In 
addition, one might argue that, the restricted definition of 
metalinguistic knowledge should be widened. 

The Grammatical Sensivitity component of aptitude should be 
complemented with measures of Inductive Language Learning 
Ability. 

The representativeness of the Lancaster pilot study results 
for undergraduate students of French more generally is 
obviously questionable. A further study would need to cover 
other institutions. 

The proposed follow-up study would allow: 

1. Systematic comparison of levels of metalinguistic knowledge 
from Bloor (1986a and c), and the results of this pilot study. 

2. Widening of the database on metalinguistic knowledge to 
cover the intakes of first-year undergraduate students of 
French in a range of institutions; 

3. Comparison of a range of . measures of linguistic ability 
with A-Level results in French and other language subjects. 

4. Relation, for a larger, fairly homogeneous sample, of 
levels ox metalinguistic knowledge to levels of accuracy in 
French, French proficiency and language aptitude. 

5. Consideration of the pedagogic ai.l curricular implications 
of the resultant findings about levels of knowledge, accuracy 
and proficiency. 
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